Fully distributed actor-critic architecture for multitask deep reinforcement learning

We propose a fully distributed actor-critic architecture, named Diff-DAC, with application to multitask reinforcement learning (MRL). During the process, agents communicate their value and policy parameters neighbours, diffusing information across network of no need for central station. Each agent can only access data from its local task, but aims learn common that performs well whole set tasks. The architecture is scalable, since computational communication cost per depends on number neighbours rather than overall agents. derive Diff-DAC duality theory provide novel insights into framework, showing it actually an instance dual ascent method. prove almost sure convergence under general assumptions hold even deep-neural approximations. For more restrictive assumptions, we also this stationary point approximation original problem. Numerical results extensions continuous control benchmarks demonstrate stabilises has regularising effect induces higher performance better generalisation properties previous architectures.